{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "DDJwQPZcupab",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# EECS 498-007/598-005 Assignment 5-2: Two-Stage Object Detector - Faster R-CNN\n",
    "\n",
    "Before we start, please put your name and UMID in following format\n",
    "\n",
    ": Firstname LASTNAME, #00000000   //   e.g.) Justin JOHNSON, #12345678"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "JSXasOiouZdl",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Your Answer:**   \n",
    "Your NAME, #XXXXXXXX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "PoRTyUc94S1a",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Two-Stage Object Detector\n",
    "In this exercise you will implement a **two-stage** object detector, based on [Faster R-CNN](https://arxiv.org/pdf/1506.01497.pdf), which consists of two modules, Region Proposal Networks (RPN) and Fast R-CNN. We will later use it to train a model that can detect objects on novel images and evaluate the detection accuracy using the classic metric mean Average Precision ([mAP](https://github.com/Cartucho/mAP))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "LfBk3NtRgqaV",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Getting Started"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "ubB_0e-UAOVK",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Setup Code\n",
    "Before getting started, we need to run some boilerplate code to set up our environment, same as previous assignments. You'll need to rerun this setup code each time you start the notebook.\n",
    "\n",
    "First, run this cell load the autoreload extension. This allows us to edit .py source files, and re-import them into the notebook for a seamless editing and debugging experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "executionInfo": {
     "elapsed": 409,
     "status": "ok",
     "timestamp": 1604336756706,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "ASkY27ZtA7Is",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "MzqbYcKdz6ew",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Google Colab Setup\n",
    "Next we need to run a few commands to set up our environment on Google Colab. If you are running this notebook on a local machine you can skip this section.\n",
    "\n",
    "Run the following cell to mount your Google Drive. Follow the link, sign in to your Google account (the same account you used to store this notebook!) and copy the authorization code into the text box that appears below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 588,
     "status": "ok",
     "timestamp": 1604336756898,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "HzRdJ3uhe1CR",
    "new_sheet": false,
    "outputId": "fa9a4e6e-6da7-426c-cdfc-7cf69b12e524",
    "run_control": {
     "read_only": false
    },
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "OvUDZWGU3VLV",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now recall the path in your Google Drive where you uploaded this notebook, fill it in below. If everything is working correctly then running the folowing cell should print the filenames from the assignment:\n",
    "\n",
    "```\n",
    "['single_stage_detector_yolo.ipynb', 'two_stage_detector_faster_rcnn.ipynb', 'eecs598', 'single_stage_detector.py', 'two_stage_detector.py', 'a5_helper.py']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 577,
     "status": "ok",
     "timestamp": 1604336756898,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "RrAX9FOLpr9k",
    "new_sheet": false,
    "outputId": "439f3a5e-a10f-4c84-ba41-97e9718cdde3",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# TODO: Fill in the Google Drive path where you uploaded the assignment\n",
    "# Example: If you create a 2020FA folder and put all the files under A5 folder, then '2020FA/A5'\n",
    "GOOGLE_DRIVE_PATH_AFTER_MYDRIVE = None\n",
    "GOOGLE_DRIVE_PATH = os.path.join('drive', 'My Drive', GOOGLE_DRIVE_PATH_AFTER_MYDRIVE)\n",
    "print(os.listdir(GOOGLE_DRIVE_PATH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "RldDumJE48pv",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Once you have successfully mounted your Google Drive and located the path to this assignment, run th following cell to allow us to import from the `.py` files of this assignment. If it works correctly, it should print the message:\n",
    "\n",
    "```\n",
    "Hello from single_stage_detector.py!\n",
    "Hello from a5_helper.py!\n",
    "```\n",
    "\n",
    "as well as the last edit time for the file `single_stage_detector.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 1081,
     "status": "ok",
     "timestamp": 1604336757413,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "pTIwSpkS495_",
    "new_sheet": false,
    "outputId": "bd251f4e-5639-412b-93ac-f0ddce7c0dc0",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(GOOGLE_DRIVE_PATH)\n",
    "\n",
    "import time, os\n",
    "os.environ[\"TZ\"] = \"US/Eastern\"\n",
    "time.tzset()\n",
    "\n",
    "from two_stage_detector import hello_two_stage_detector\n",
    "hello_two_stage_detector()\n",
    "\n",
    "from a5_helper import hello_helper\n",
    "hello_helper()\n",
    "\n",
    "two_stage_detector_path = os.path.join(GOOGLE_DRIVE_PATH, 'two_stage_detector.py')\n",
    "two_stage_detector_edit_time = time.ctime(os.path.getmtime(two_stage_detector_path))\n",
    "print('two_stage_detector.py last edited on %s' % two_stage_detector_edit_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "GWP1vCGL5Eca",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Load several useful packages that are used in this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5651,
     "status": "ok",
     "timestamp": 1604336761995,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "CwVZ26yM5G8U",
    "new_sheet": false,
    "outputId": "a76987a8-55fc-438f-8859-a23aed779c54",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torchvision\n",
    "import statistics\n",
    "import random\n",
    "import time\n",
    "import math\n",
    "import numpy as np\n",
    "import cv2\n",
    "import copy\n",
    "import shutil\n",
    "import os\n",
    "import json\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from eecs598 import reset_seed, Solver\n",
    "from eecs598.grad import rel_error\n",
    "\n",
    "from a5_helper import *\n",
    "\n",
    "# for plotting\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['font.size'] = 16\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for mAP evaluation\n",
    "!rm -rf mAP\n",
    "!git clone https://github.com/Cartucho/mAP.git\n",
    "!rm -rf mAP/input/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "x7poKGI35JZY",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We will use GPUs to accelerate our computation in this notebook. Run the following to make sure GPUs are enabled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5644,
     "status": "ok",
     "timestamp": 1604336761999,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "Vw3wIuCu5LnU",
    "new_sheet": false,
    "outputId": "0c522ba2-0912-4e74-b895-7b491a4eddbf",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "if torch.cuda.is_available:\n",
    "  print('Good to go!')\n",
    "else:\n",
    "  print('Please set GPU via Edit -> Notebook Settings.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "0Saw9jGNm-9-",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Import functions from previous notebook\n",
    "We will re-use some pieces of code that you implemented in the previous notebook. The functions needed are `GenerateAnchor`, `GenereateProposal`, and `IoU`. Make sure to implement them before working on this notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5633,
     "status": "ok",
     "timestamp": 1604336761999,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "Xq5yHDeuklId",
    "new_sheet": false,
    "outputId": "e2e9a686-6c18-4953-c283-8a62120c46a9",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from single_stage_detector import GenerateAnchor, GenerateProposal, IoU\n",
    "\n",
    "print('Import successful!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "MjJ3uyYBg3Lw",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Load PASCAL VOC 2007 data\n",
    "As in the previous notebook, we will use the PASCAL VOC 2007 dataset to train our object detection system.\n",
    "\n",
    "As in the previous notebook, we will subsample the dataset and wrap it in a DataLoader that can form minibatches for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 32177,
     "status": "ok",
     "timestamp": 1604336788554,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "MmEP5KQJzk0d",
    "new_sheet": false,
    "outputId": "1f13cb78-175c-40c0-d6b3-ea22bcf990de",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# uncomment below to use the mirror link if the original link is broken and move it to GOOGLE_DRIVE_PATH\n",
    "# !wget https://web.eecs.umich.edu/~justincj/data/VOCtrainval_06-Nov-2007.tar\n",
    "# shutil.move('VOCtrainval_06-Nov-2007.tar', os.path.join(GOOGLE_DRIVE_PATH, 'VOCtrainval_06-Nov-2007.tar'))  \n",
    "\n",
    "train_dataset = get_pascal_voc2007_data(GOOGLE_DRIVE_PATH, 'train')\n",
    "val_dataset = get_pascal_voc2007_data(GOOGLE_DRIVE_PATH, 'val')\n",
    "\n",
    "train_dataset = torch.utils.data.Subset(train_dataset, torch.arange(0, 2500)) # use 2500 samples for training\n",
    "train_loader = pascal_voc2007_loader(train_dataset, 10)\n",
    "val_loader = pascal_voc2007_loader(val_dataset, 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 36636,
     "status": "ok",
     "timestamp": 1604336793026,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "nZVYFJD32I_l",
    "new_sheet": false,
    "outputId": "350657f7-8eed-44fb-ac74-a02687e2eeea",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "train_loader_iter = iter(train_loader)\n",
    "img, ann, _, _, _ = train_loader_iter.next()\n",
    "\n",
    "print('Resized train images shape: ', img[0].shape)\n",
    "print('Padded annotation tensor shape: ', ann[0].shape)\n",
    "print(ann[0])\n",
    "print('Each row in the annotation tensor indicates (x_tl, y_tl, x_br, y_br, class).')\n",
    "print('Padded with bounding boxes (-1, -1, -1, -1, -1) to enable batch loading. (You may need to run a few times to see the paddings)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "X4WmocEyiXWa",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Visualize PASCAL VOC 2007\n",
    "Sample a couple of images and GT boxes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 36626,
     "status": "ok",
     "timestamp": 1604336793027,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "ld1s28Z4fyL5",
    "new_sheet": false,
    "outputId": "b2c9a1a3-e4ea-4d15-8a05-22c45f80323d",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# default examples for visualization\n",
    "reset_seed(0)\n",
    "batch_size = 3\n",
    "sampled_idx = torch.linspace(0, len(train_dataset)-1, steps=batch_size).long()\n",
    "\n",
    "# get the size of each image first\n",
    "h_list = []\n",
    "w_list = []\n",
    "img_list = [] # list of images\n",
    "MAX_NUM_BBOX = 40\n",
    "box_list = torch.LongTensor(batch_size, MAX_NUM_BBOX, 5).fill_(-1) # PADDED GT boxes\n",
    "\n",
    "for idx, i in enumerate(sampled_idx):\n",
    "  # hack to get the original image so we don't have to load from local again...\n",
    "  img, ann = train_dataset.__getitem__(i)\n",
    "  img_list.append(img)\n",
    "\n",
    "  all_bbox = ann['annotation']['object']\n",
    "  if type(all_bbox) == dict:\n",
    "    all_bbox = [all_bbox]\n",
    "  for bbox_idx, one_bbox in enumerate(all_bbox):\n",
    "    bbox = one_bbox['bndbox']\n",
    "    obj_cls = one_bbox['name']\n",
    "    box_list[idx][bbox_idx] = torch.LongTensor([int(bbox['xmin']), int(bbox['ymin']),\n",
    "      int(bbox['xmax']), int(bbox['ymax']), class_to_idx[obj_cls]])\n",
    "\n",
    "  # get sizes\n",
    "  img = np.array(img)\n",
    "  w_list.append(img.shape[1])\n",
    "  h_list.append(img.shape[0])\n",
    "\n",
    "w_list = torch.tensor(w_list, dtype=torch.float32, device='cuda')\n",
    "h_list = torch.tensor(h_list, dtype=torch.float32, device='cuda')\n",
    "box_list = torch.tensor(box_list, dtype=torch.float32, device='cuda')\n",
    "resized_box_list = coord_trans(box_list, w_list, h_list, mode='p2a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 37783,
     "status": "ok",
     "timestamp": 1604336794196,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "v04D-gwEiWqY",
    "new_sheet": false,
    "outputId": "04826e4d-69cb-4b61-8b17-ae5b4a17cee7",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualize GT boxes\n",
    "for i in range(len(img_list)):\n",
    "  valid_box = sum([1 if j != -1 else 0 for j in box_list[i][:, 0]])\n",
    "  eecs598.vis.detection_visualizer(img_list[i], idx_to_class, box_list[i][:valid_box])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "oRc7P-RvRZGZ",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Region Proposal Networks (RPN)\n",
    "The first stage in a Faster R-CNN object detector is the *Region Proposal Network (RPN)*. The RPN classifies a set of anchors as either containing an object or not, and also regresses from the position of the anchor box to a region proposal.\n",
    "\n",
    "The RPN is very similar to the single-stage detector we built in the previous notebook, except that it will not predict classification scores. We can therefore reuse many of the functions from the previous notebook in order to implement the RPN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "etBYc7rbj35F",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Anchor\n",
    "We will use the exact same set of anchors as in the single-stage detector from the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 37772,
     "status": "ok",
     "timestamp": 1604336794196,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "O5w-EUJekJj-",
    "new_sheet": false,
    "outputId": "49162f9d-f073-4d9d-e9b8-541c6ae3f981",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Declare variables for anchor priors, a Ax2 Tensor where A is the number of anchors.\n",
    "# Hand-picked, same as our two-stage detector.\n",
    "anchor_list = torch.tensor([[1, 1], [2, 2], [3, 3], [4, 4], [5, 5], [2, 3], [3, 2], [3, 5], [5, 3]], \n",
    "                           dtype=torch.float32, device='cuda')\n",
    "print(anchor_list.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "7iHgTW0KTj13",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Activated (positive) and negative anchors\n",
    "When training the RPN, we compare the anchor boxes with the ground-truth boxes in order to determine a ground-truth label for the anchor boxes -- should each anchor predict object or background?\n",
    "\n",
    "We assign a positive label to two kinds of anchors:\n",
    "\n",
    "(i) the anchor/anchors with the highest Intersection-overUnion (IoU) overlap with a ground-truth box, or\n",
    "\n",
    "(ii) an anchor that has an IoU overlap higher than 0.7 with any ground-truth box. Note that a single ground-truth box may assign positive labels to multiple anchors.\n",
    "\n",
    "Usually the second condition is sufficient to determine the positive samples; but we still adopt the first condition for the reason that in some rare cases the second condition may find no positive sample.\n",
    "\n",
    "We assign a negative label to a non-positive anchor if its IoU ratio is lower than 0.3 for all ground-truth boxes. Anchors that are neither positive nor negative do not contribute to the training objective"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "oalEB2-Pa1zQ",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We can implement anchor generation and matching to ground-truth by reusing the `GenerateGrid`, `GenerateAnchor`, `IoU`, and `ReferenceOnActivatedAnchors` functions from the previous notebook.\n",
    "\n",
    "Run the following to check the implementation from A5-1 (with your IoU function). You should see errors on the order of 1e-7 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 39235,
     "status": "ok",
     "timestamp": 1604336795669,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "fK_USCuaXSzh",
    "new_sheet": false,
    "outputId": "eda44250-2f2f-4cbd-8e53-aec8d452b800",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "reset_seed(0)\n",
    "\n",
    "grid_list = GenerateGrid(w_list.shape[0])\n",
    "anc_list = GenerateAnchor(anchor_list, grid_list)\n",
    "iou_mat = IoU(anc_list, resized_box_list)\n",
    "activated_anc_ind, negative_anc_ind, GT_conf_scores, GT_offsets, GT_class, \\\n",
    "  activated_anc_coord, negative_anc_coord = ReferenceOnActivatedAnchors(anc_list, resized_box_list, grid_list, iou_mat)\n",
    "\n",
    "expected_GT_conf_scores = torch.tensor([0.74538743, 0.72793430, 0.71128041, 0.70029843,\n",
    "                                        0.75670898, 0.76044953, 0.37116671, 0.37116671],\n",
    "                                       dtype=torch.float32, device='cuda')\n",
    "expected_GT_offsets = torch.tensor([[ 0.01633334,  0.11911901, -0.09431065,  0.19244696],\n",
    "                                    [-0.03675002,  0.09324861, -0.00250307,  0.25213102],\n",
    "                                    [-0.03675002, -0.15675139, -0.00250307,  0.25213102],\n",
    "                                    [-0.02940002,  0.07459889, -0.22564663,  0.02898745],\n",
    "                                    [ 0.11879997,  0.03208542,  0.20863886, -0.07974572],\n",
    "                                    [-0.08120003,  0.03208542,  0.20863886, -0.07974572],\n",
    "                                    [ 0.07699990,  0.28533328, -0.03459148, -0.86750042],\n",
    "                                    [ 0.07699990, -0.21466672, -0.03459148, -0.86750042]],\n",
    "                                   dtype=torch.float32, device='cuda')\n",
    "expected_GT_class = torch.tensor([ 6,  7,  7,  7, 19, 19,  6,  6], dtype=torch.int64, device='cuda')\n",
    "print('conf scores error: ', rel_error(GT_conf_scores, expected_GT_conf_scores))\n",
    "print('offsets error: ', rel_error(GT_offsets, expected_GT_offsets))\n",
    "print('class prob error: ', rel_error(GT_class, expected_GT_class))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 40245,
     "status": "ok",
     "timestamp": 1604336796690,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "wPvX4TrgaLD8",
    "new_sheet": false,
    "outputId": "8ea32253-e09d-4a92-c8b3-b27f816969cd",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualize the activated anchors\n",
    "anc_per_img = torch.prod(torch.tensor(anc_list.shape[1:-1]))\n",
    "\n",
    "print('*'*80)\n",
    "print('Activated (positive) anchors:')\n",
    "for img, bbox, idx in zip(img_list, box_list, torch.arange(box_list.shape[0])):\n",
    "  anc_ind_in_img = (activated_anc_ind >= idx * anc_per_img) & (activated_anc_ind < (idx+1) * anc_per_img)\n",
    "  print('{} activated anchors!'.format(torch.sum(anc_ind_in_img)))\n",
    "  eecs598.vis.detection_visualizer(img, idx_to_class, bbox[:, :4], coord_trans(activated_anc_coord[anc_ind_in_img], w_list[idx], h_list[idx]))\n",
    "\n",
    "print('*'*80)\n",
    "print('Negative anchors:')\n",
    "for img, bbox, idx in zip(img_list, box_list, torch.arange(box_list.shape[0])):\n",
    "  anc_ind_in_img = (negative_anc_ind >= idx * anc_per_img) & (negative_anc_ind < (idx+1) * anc_per_img)\n",
    "  print('{} negative anchors!'.format(torch.sum(anc_ind_in_img)))\n",
    "  eecs598.vis.detection_visualizer(img, idx_to_class, bbox[:, :4], coord_trans(negative_anc_coord[anc_ind_in_img], w_list[idx], h_list[idx]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "XW_Zek3_dgfF",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Proposal module\n",
    "Similar to the Prediction Networks in A5-1, but for RPN you only need to predict the object proposal scores (from the *cls* layer) and bounding box offsets (from the *reg* layer), all of which are class-agnostic.\n",
    "\n",
    "![pred_scores2](https://miro.medium.com/max/918/1*wB3ctS9WGNmw6pP_kjLjgg.png)\n",
    "\n",
    "Note that here $k$ is essentially $A$. Image credit: Ren et al, \"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks\", NeurIPS 2015, https://arxiv.org/abs/1506.01497\n",
    "\n",
    "Implement the forward pass of `ProposalNetwork` in `two_stage_detector.py`. Run the following to check your implementation. You should see errors on the order of 1e-7 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 40233,
     "status": "ok",
     "timestamp": 1604336796690,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "MX2JCaOf0768",
    "new_sheet": false,
    "outputId": "b550a5c7-75af-4f85-da5a-c1ff684de6af",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from two_stage_detector import ProposalModule\n",
    "\n",
    "# sanity check\n",
    "reset_seed(0)\n",
    "prop_module = ProposalModule(1280, drop_ratio=0).to(dtype=torch.float32, device='cuda')\n",
    "features = torch.linspace(-10., 10., steps=3*1280*7*7, dtype=torch.float32, device='cuda').view(3, 1280, 7, 7)\n",
    "conf_scores, offsets, proposals = prop_module(features, activated_anc_coord, \\\n",
    "              pos_anchor_idx=activated_anc_ind, neg_anchor_idx=negative_anc_ind)\n",
    "\n",
    "expected_conf_scores = torch.tensor([[-0.50843990,  2.62025023],\n",
    "                                     [-0.55775326, -0.29983672],\n",
    "                                     [-0.55796617, -0.30000290],\n",
    "                                     [ 0.17819080, -0.42211828],\n",
    "                                     [-0.51439995, -0.47708601],\n",
    "                                     [-0.51439744, -0.47703803],\n",
    "                                     [ 0.63225138,  2.71269488],\n",
    "                                     [ 0.63224381,  2.71290708]],\n",
    "                                    dtype=torch.float32, device='cuda')\n",
    "expected_offsets = torch.tensor([[ 1.62754285,  1.35253453, -1.85451591, -1.77882397],\n",
    "                                 [-0.33651856, -0.14402901, -0.07458937, -0.27201492],\n",
    "                                 [-0.33671042, -0.14398587, -0.07479107, -0.27199429],\n",
    "                                 [ 0.06847382,  0.21062726,  0.09334904, -0.02446130],\n",
    "                                 [ 0.16506940, -0.30296192,  0.29626080,  0.32173073],\n",
    "                                 [ 0.16507357, -0.30302414,  0.29625297,  0.32169008],\n",
    "                                 [ 1.59992146, -0.75236654,  1.66449440,  2.05138564],\n",
    "                                 [ 1.60008609, -0.75249159,  1.66474164,  2.05162382]], \n",
    "                                dtype=torch.float32, device='cuda')\n",
    "\n",
    "print('conf scores error: ', rel_error(conf_scores[:8], expected_conf_scores))\n",
    "print('offsets error: ', rel_error(offsets, expected_offsets))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "dlO2IUCnt4zu",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Loss Function\n",
    "The confidence score regression loss is for both activated/negative anchors while the bounding box regression loss loss is for activated anchors only. \n",
    "\n",
    "\n",
    "The loss functions `ConfScoreRegression` and `BboxRegression` are both found in `two_stage_detection.py`. Run the following to check your implementation. You should see errors on the order of 1e-7 or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 40224,
     "status": "ok",
     "timestamp": 1604336796691,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "2eSleGX9yTeo",
    "new_sheet": false,
    "outputId": "ccd06602-78a2-43fb-ee01-a5431da59393",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from two_stage_detector import ConfScoreRegression, BboxRegression\n",
    "\n",
    "conf_loss = ConfScoreRegression(conf_scores, features.shape[0])\n",
    "reg_loss = BboxRegression(offsets, GT_offsets, features.shape[0])\n",
    "print('conf loss: {:.4f}, reg loss: {:.4f}'.format(conf_loss, reg_loss))\n",
    "\n",
    "loss_all = torch.tensor([conf_loss.data, reg_loss.data], dtype=torch.float32, device='cuda')\n",
    "expected_loss = torch.tensor([8.55673981, 5.10593748], dtype=torch.float32, device='cuda')\n",
    "\n",
    "print('loss error: ', rel_error(loss_all, expected_loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "AiPfXUHPupDE",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## RPN module\n",
    "Implement Region Proposal Network. Should resemble the `SingleStageDetector` module from A5-1, but without the class prediction. Implement RPN in `two_stage_detector.py`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "dmukyW51hRHa",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## RPN solver\n",
    "In Faster R-CNN, the RPN is trained jointly with the second-stage network. However, to test our RPN implementation, we will first train just the RPN; this is basically a class-agnostic single-stage detector, that only classifies regions as object or background."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "executionInfo": {
     "elapsed": 40212,
     "status": "ok",
     "timestamp": 1604336796691,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "6puWIsYQhQkW",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "RPNSolver = DetectionSolver # the same solver as in YOLO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "xVQPPDVchAGQ",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## RPN - Overfit small data\n",
    "First we will overfit the RPN on a small subset of the PASCAL VOC 2007 dataset. After training you should see a loss around or less than 3.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 61721,
     "status": "ok",
     "timestamp": 1604336818203,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "YTObddiog9wJ",
    "new_sheet": false,
    "outputId": "aa4bcb05-2b38-4006-8f89-ca4a9af3835d",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from two_stage_detector import RPN\n",
    "\n",
    "# monitor the training loss\n",
    "num_sample = 10\n",
    "small_dataset = torch.utils.data.Subset(train_dataset, torch.linspace(0, len(train_dataset)-1, steps=num_sample).long())\n",
    "small_train_loader = pascal_voc2007_loader(small_dataset, 10) # a new loader\n",
    "\n",
    "for lr in [1e-3]:\n",
    "  print('lr: ', lr)\n",
    "  rpn = RPN()\n",
    "  RPNSolver(rpn, small_train_loader, learning_rate=lr, num_epochs=200, dtype=torch.float32, device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "AOIazbsrFEVc",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## RPN - Inference\n",
    "We will now visualize the predicted boxes from the RPN that we overfit to a small training sample. We will reuse the `DetectionInference` function from the previous notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "executionInfo": {
     "elapsed": 61828,
     "status": "ok",
     "timestamp": 1604336818313,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "o2wxjsnYFDrw",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "RPNInference = DetectionInference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 224,
     "output_embedded_package_id": "1N0QWFnsT-hcrCXSjOW6AYjQFwhpfCUGU"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 66700,
     "status": "ok",
     "timestamp": 1604336823188,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "94vbKR2JJ-Mq",
    "new_sheet": false,
    "outputId": "0f5e47fe-c7a7-47a3-9512-83df5d02cd6c",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualize the output from the overfitted model on small dataset\n",
    "# the bounding boxes should be really accurate\n",
    "# ignore the dummy object class (in blue) as RPN does not output class!\n",
    "RPNInference(rpn, small_train_loader, small_dataset, idx_to_class, thresh=0.8, nms_thresh=0.3, device='cuda', dtype=torch.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "jKjv6JLMRj7s",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Faster R-CNN\n",
    "We have implemented the first half of Faster R-CNN, i.e., RPN, which is class-agnostic. Here, we briefly describe the second half Fast R-CNN.\n",
    "\n",
    "Given the proposals or region of interests (RoI) from RPN, we warp each region from CNN activation map to a fixed size 2x2 by using [RoI Align](https://arxiv.org/pdf/1703.06870.pdf). Essentially, the RoI feature is determined by bilinear interpolation over the CNN activation map. We meanpool the RoI feature over the spatial dimension (2x2).\n",
    "\n",
    "Finally, we classify the meanpooled RoI feature into class probabilities.\n",
    "\n",
    "For simplicity, our two-stage detector here differs from a full Faster R-CNN system in a few aspects.\n",
    "1. In a full implementation, the second stage of the network would predict a second set of offsets to transform the region proposal into a final predicted object bounding box. However we omit this for simplicity.\n",
    "2. In a full implementation, the second stage of the network should be able to reject negative boxes -- in other words, if we want to predict C different object categories then the final classification layer of the second stage would predict a distribution over C+1 categories, with an extra one for background. We omit this, as it requires extra bookeeping in the second stage about which proposals are positive / negative; so for simplicity our second stage will only predict a distribution over C categories, and we will assume that the RPN has filtered out all background regions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "hdqlAcP6kRvA",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## RoI Align\n",
    "We will use the `roi_align` function from `torchvision`. Usage see https://pytorch.org/docs/stable/torchvision/ops.html#torchvision.ops.roi_align"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "RFZ49wox4MYn",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Overfit small data\n",
    "\n",
    "Implement the TwoStageDetector network in `two_stage_detector.py`. You can then test your implementation by overfitting the full Faster R-CNN network on a small subset of the training data. After training you should see a final loss less than 4.0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 87583,
     "status": "ok",
     "timestamp": 1604336844074,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "WbxeAJq0zc3F",
    "new_sheet": false,
    "outputId": "6a9c8d88-29c7-4a83-db92-8e75b2522d3d",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from two_stage_detector import TwoStageDetector\n",
    "# monitor the training loss\n",
    "\n",
    "lr = 1e-3\n",
    "detector = TwoStageDetector().to(dtype=torch.float32, device='cuda')\n",
    "DetectionSolver(detector, small_train_loader, learning_rate=lr, num_epochs=200, dtype=torch.float32, device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "_SWA1DbG47ln",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 224,
     "output_embedded_package_id": "11365lX4HCQuMWjqvDNZLW_bbkmjBYgPa"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 92245,
     "status": "ok",
     "timestamp": 1604336848739,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "gp_Hmt-Km5bl",
    "new_sheet": false,
    "outputId": "b2fe0c8e-fb0e-4151-8021-ca7637c95b2a",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualize the output from the overfitted model on small dataset\n",
    "# the bounding boxes should be really accurate\n",
    "DetectionInference(detector, small_train_loader, small_dataset, idx_to_class, thresh=0.8, nms_thresh=0.3, dtype=torch.float32, device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "sr7wNngy4oZf",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Train a net\n",
    "Now it's time to train the full Faster R-CNN model on a larger subset of the the training data. We will train for 50 epochs; this should take about 30 minutes on a collab instance. You should see a total loss less than 3.0.\n",
    "\n",
    "(Optional) If you train the model longer (e.g., 100 epochs), you should see a better mAP. But make sure you revert the code back for grading purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 1245035,
     "status": "ok",
     "timestamp": 1604338388407,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "X1k1rx1f4sTE",
    "new_sheet": false,
    "outputId": "53016a93-7d20-4bd1-e53c-5e2ea819af05",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# monitor the training loss\n",
    "train_loader = pascal_voc2007_loader(train_dataset, batch_size=100) # a new loader\n",
    "\n",
    "num_epochs = 50\n",
    "lr = 5e-3\n",
    "frcnn_detector = TwoStageDetector().to(dtype=torch.float32, device='cuda')\n",
    "DetectionSolver(frcnn_detector, train_loader, learning_rate=lr, num_epochs=num_epochs, dtype=torch.float32, device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "cnwjLTJvxx0r",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "After you've trained your model for 50 epochs, save it for submission. We reload it the weights and check the model accuracy as a sanity check.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 18682,
     "status": "ok",
     "timestamp": 1604338647071,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "qERg85Ip7hwu",
    "new_sheet": false,
    "outputId": "e9491e80-a841-41ee-8c04-0c790ad1d32f",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "weights_path = os.path.join(GOOGLE_DRIVE_PATH, \"frcnn_detector.pt\")\n",
    "torch.save(frcnn_detector.state_dict(), weights_path) # uncomment to save your checkpoint\n",
    "\n",
    "frcnn_detector = TwoStageDetector().to(dtype=torch.float32, device='cuda')\n",
    "frcnn_detector.load_state_dict(torch.load(weights_path)) # uncomment to load your previous checkpoint\n",
    "DetectionEvaluater(frcnn_detector, train_loader, dtype=torch.float32, device='cuda')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "KhWZT-ztEaqm",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "output_embedded_package_id": "1ichXEDYlWPmfoeTM9K6emTm7xyEiop4b"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 4557,
     "status": "ok",
     "timestamp": 1604338499939,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "J7ArGiLTnHta",
    "new_sheet": false,
    "outputId": "ba366134-e564-40db-c4ec-f6b18c0abb08",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# visualize the same output from the model trained on the entire training set\n",
    "# some bounding boxes might not make sense\n",
    "DetectionInference(frcnn_detector, small_train_loader, small_dataset, idx_to_class, device='cuda', dtype=torch.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "ETU6ev7aydIY",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Evaluation\n",
    "Compute mean Average Precision (mAP). Introduction on mAP see lecture slides (p46-57): https://web.eecs.umich.edu/~justincj/slides/eecs498/498_FA2019_lecture15.pdf\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "8iGOlxhyvO3z",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Run the following to evaluate your detector on the PASCAL VOC validation set. You should see mAP at around 16% or above.\n",
    "\n",
    "(Optional) If you train the model longer (e.g., 100 epochs), you should see a better mAP. But make sure you revert the code back for grading purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 39181,
     "status": "ok",
     "timestamp": 1604338539394,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "FvDb7uwqyhAK",
    "new_sheet": false,
    "outputId": "f39788ae-e90a-418c-e625-bd52d956703c",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "DetectionInference(frcnn_detector, val_loader, val_dataset, idx_to_class, output_dir='mAP/input', thresh=0.8, nms_thresh=0.3, device='cuda', dtype=torch.float32)\n",
    "# DetectionInference(frcnn_detector, train_loader, train_dataset, idx_to_class, output_dir='mAP/input', thresh=0.8, nms_thresh=0.3, device='cuda', dtype=torch.float32) # uncomment to see training mAP\n",
    "!cd mAP && python main.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "4u69TlnhucrR",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Submit Your Work\n",
    "After completing both notebooks for this assignment (`single_stage_detector_yolo.ipynb` and this notebook, `two_stage_detector_faster_rcnn.ipynb`), run the following cell to create a `.zip` file for you to download and turn in. **Please MANUALLY SAVE every `*.ipynb` and `*.py` files before executing the following cell:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "executionInfo": {
     "elapsed": 220660,
     "status": "aborted",
     "timestamp": 1604336977202,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 300
    },
    "id": "e6vziUpSuqLY",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from eecs598.submit import make_a5_submission\n",
    "\n",
    "# TODO: Replace these with your actual uniquename and umid\n",
    "uniquename = None\n",
    "umid = None\n",
    "make_a5_submission(GOOGLE_DRIVE_PATH, uniquename, umid)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "xVQPPDVchAGQ",
    "AOIazbsrFEVc",
    "RFZ49wox4MYn"
   ],
   "name": "two_stage_detector_faster_rcnn.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
